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MDA  W&A  Definitions 


Three  interrelated  but  distinct  processes  that  gather  and  evaluate  evidence  to  determine, 
based  on  the  simulation’s  intended  use,  the  simulation’s  capabilities,  limitations,  and 
performance  relative  to  the  real-world  objects  it  simulates  are: 


-  Verification: 

-  Process  of  determining  that  a  model  implementation  and  its  associated  data 
accurately  represent  the  developer's  conceptual  description  and  specifications 

-  Validation: 

-  Process  of  determining  the  degree  to  which  a  model  and  its  associated  data  are 
an  accurate  representation  of  the  real-world  from  the  perspective  of  the  intended 
uses  of  the  model 

-  Techniques  are  not  limited  to  comparison  of  simulation  results  with  test  data. 
Among  the  other  techniques  employed  are  data  validation,  sensitivity  analyses 
to  evaluate  input/output  relationships,  comparison  with  other  models  and 
simulations  known  (or  assumed)  to  have  validity  in  the  operating  range  required 
(benchmarking),  and  the  results  of  SME  reviews  of  M&S  outputs  (face 
validation) 

-  One  method  of  validation  is  anchoring 

-  Accreditation: 

-  The  official  certification  that  a  model,  simulation,  or  federation  of  models  and 
simulations  and  its  associated  data  are  acceptable  for  use  for  a  specific  purpose 


Source:  MDA  Directive  8315.aa,  June  2007  -  M&S  Recommended  Practices  Guide  (RPG) 
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System-Level  Post-Flight  Reconstruction 
and  Anchoring  Definitions 

-System-Level  Post-Flight  Reconstruction  (PFR): 

»  Manually  recreate  and  run  a  past  flight  test  scenario  in  a  test 
venue  performing  system-level  comparative  analysis  of  the  real- 
world  performance  to  the  output  of  the  test  venue  assessing  the 
results  and  determining  if  system-level  anomalies  exist  in  the 
Models  &  Simulations  (M&S) 


-System-Level  Anchoring  (SLA): 

»  Perform  root  cause  analysis  of  the  system-level  anomalies  found  in 
the  PFR;  generate,  test  and  implement  M&S  improvements  to 
address  anomalies 
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System  of  Systems 


Independent  systems  integrated  into  the  larger  System  of  systems 
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Foundation  of  System  PFR 


System-Level  Validation  is  built  on  individual  Element  Validation 
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M&S  System-Level  PFR/Anchoring  Process 
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Factors  Influencing  the 
M&S  System-Level  PFR  and  SLA  Process 


•  Executive-level  management  requirements  (i.e.  timeframe,  perform 
PFR  only  or  follow  the  process  through  to  M&S  improvement 
(Anchoring),  etc.) 

•  Number  of  parameters  to  be  validated 

•  PFR  lab  location  and  schedule 

•  Element  funding  and  allocation 

•  Integrated  or  distributed  lab  configuration 

•  Hardware-in-the-loop  (HWIL)  and/or  end-to-end  digital  simulation 

•  Required  organizations  and  resources 

•  Intended  uses 

•  Who  will  generate  the  threat  input 

•  Which  signature  package  will  be  used 

•  S/W  configuration  change  cycle 

•  S/W  configuration  comparisons 
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System-Level  Analysis  (SLA) 


•  Verification 

-  Was  the  Test  able  to  show  Objectives  were  met  or  not  met? 

•  Validation 

-  How  close  do  the  models  match  real-world? 

•  Threat  /  Target 

-  Does  radar  model  response  reflect  real-world  play  backs? 

-  Was  Threat  presented  to  Elements  consistently? 

-  Trajectory  (Pos,  Vel,  Acc,  Body  Axis,  Body  rates)  comparisons 

-  Signature  comparisons 

•  Environment 

-  Is  the  modeled  Environment  representative  of  real-world? 

-  Modeled  weather  comparisons  to  real  world  observations 

-  Degradations  of  Element  performance  due  to  weather  conditions 

•  Interfaces  between  different  Elements  (not  within  a  single  Element) 

-  Is  the  modeled  architecture  appropriate  for  the  test? 

-  Does  the  message  flow  /  interaction  between  Elements  match  the  real-world  test? 

-  Number  of  Messages  in/out,  Types  of  Messages,  Message  Content,  Latency 

•  System-Level  Key  Functions  and  Data  Elements 

-  Once  defined  need  to  be  applied  to  PFR  analysis 

•  Kill  Chain  Functions  applicable  to  Entire  System 

-  Detect,  Track,  Discriminate,  Engage,  Negate,  Assess 

-  Did  the  model  perform  these  functions  like  the  real-world  system? 
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Anchoring  Methodologies  Considered 


Case  #1:  Point-to-Point  Comparison  Using  GT 


Case  #2:  Statistical  Comparison  Using 

Digital  Representation 


♦ 


Legend 

•  Real  World 
A  Ground  Test 


Pros: 

•  Quick  look  at  data 

•  More  timely  analysis 

•  Analysis  tool  is  not  required 

Cons: 

•  Sample  data  set  is  not  statistical  and  may 
represent  the  full  sample  space 

•  Lower  confidence  in  the  assessment  or 
assessment  may  not  be  possible 

•  Tolerance  may  not  be  accurate 


not 


Legend 

•  Real  World 

jk.  Digital  Representation 


Pros: 

•  Sample  data  set  is  statistical 

•  Higher  confidence  in  the  assessment 

•  More  accurate  root  cause  analysis 

Cons: 

•  Not  all  models  can  emulate  a  FT  exactly 

•  Analysis  tool  is  required 

•  More  time  consuming  to  analyze  data 


Case  #3:  Combination  of  Case  1  and  2 


Pros: 

•  Sample  data  set  is  statistical 

•  Highest  confidence  in  the  assessment 

•  Faster  root  cause  analysis 

•  More  thorough  analysis 

Cons: 

•  Not  all  models  can  emulate  a  FT  exactly 

•  Analysis  tool  is  required 

•  More  time  consuming  to  analyze  data 
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Path  Forward 


*  Case  #3 

-  Prove  &  develop  the  process  utilizing  available  HWIL 
test  data  and  a  known  Digital  Representation 

*  Where  we  are: 

-  Develop  and  prove  the  process  in  2008  prior  to  the 
2009  campaign 

*  Where  we  are  going: 

-  Planning  2  HWIL  and  1  End-to-end  digital  simulation 
SPFR/SLA  efforts  in  2009 

-  Using  2008  lessons  learned  to  refine  process 
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